Anisotropic Extinction Distortion of the Galaxy Correlation Function 
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Similar to the magnification of the galaxies' fluxes by gravitational lensing, the extinction of the 
fluxes by comic dust, whose existence is recently detected by also modify the distribution of a 
flux-selected galaxy sample. We study the anisotropic distortion by dust extinction to the 3D galaxy 
correlation function, including magnification bias and redshift distortion at the same time. We find 
the extinction distortion is most significant along the line of sight and at large separations, similar 
to that by magnification bias. The correction from dust extinction is negative except at sufficiently 
large transverse separations, which is almost always opposite to that from magnification bias (we 
consider a number count slope s > 0.4). Hence, the distortions from these two effects tend to reduce 
each other. At low z { <, 1), the distortion by extinction is stronger than that by magnification 
bias, but at high z, the reverse holds. We also study how dust extinction affects probes in real space 
of the baryon acoustic oscillations (BAO) and the linear redshift distortion parameter /3. We find 
its effect on BAO is negligible. However, it introduces a positive scale-dependent correction to /? 
that can be as large as a few percent. At the same time, we also find a negative scale-dependent 
correction from magnification bias, which is up to percent level at low z, but to ~ 40% at high z. 
These corrections are non-negligible for precision cosmology, and should be considered when testing 
General Relativity through the scale-dependence of /3. 
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I. INTRODUCTION 



Galaxy redshift surveys provide a S-dimensional map 
of the universe's large scale structure (LSS), and hence 
serve as important probes of cosmology. The simplest 
statistic obtained from a redshift survey is the two point 
galaxy correlation function, or its Fourier transform, the 
galaxy power spectrum. Both quantities have been mea- 
sured to good accuracy by recent large redshift surveys 
such as the 2dF Galaxy Redshift Survey (2dFGRS) [3,[| 
and the Sloan Digital Sky Survey (SDSS) H, and the re- 
sults have been used to put constraints on cosmological 
parameters, e.g., [5l-[l0| by using the correlation function 
and [Tll-[T9| by using the power spectrum. 

It is known that the measurements of these two quan- 
tities are subject to several effects, which distort them to 
be anisotropic, i.e. the measured signal depends on the 
orientation of the separation or wave vector. One effect 
comes from the possibly wrong cosmology used to con- 
vert the measured coordinates, i.e. redshifts and angular 
positions, to the comoving ones — the Alcock-Paczynski 
effect (|20l-|24|). A second one comes from the peculiar 
velocities of the galaxies that introduce uncertainties in 
the interpretation from redshift to comoving distance — 
the redshift distortion f j25l - [28j ). A third effect is caused 
by gravitational lensing mainly through magnification 
of the galaxies' fluxes (for flux-sclccted galaxy samples) 
and changes to the apparent angular separations between 
galaxies — the magnification bias |29l - t36| . 



For measurements made with galaxy samples that are 
selected by the galaxy's flux, a fourth anisotropic effect 
would arise from the existence of cosmic dust, which 
causes extinction in the fluxes of the galaxies. By cosmic 
dust we mean dust which is correlated with galaxies, but 
which may or may not reside in galaxy haloes. In a flux 
selected galaxy sample, such cosmic dust would modulate 
the galaxy density field - fewer galaxies behind regions 
of higher extinction. Moreover, the effect is anisotropic 
in 3D. Recall that a two-point correlation measurement 
is essentially a measurement of pair counts. At a given 
separation, pairs of galaxies that are aligned close to 
the line of sight suffer more of an extinction effect - i.e. 
dust correlated with the foreground galaxies dims (and 
removes from sample) the background galaxies - while 
pairs of galaxies oriented transverse to the line of sight 
are less susceptible. This is much like how gravitational 
lensing or magnification bias introduces an anisotropy to 
the galaxy correlation function or power spectrum [3J- 
[36j . The difference is that gravitational lensing by the 
foreground galaxies generally brightens the background 
galaxies, thus adding them to one's sample, and it also 
causes an overall geometrical stretching which dilutes the 
apparent number density. As we will see, the precise 
shapes of these two anisotropics are different. We also 
notice here that there could be other anisotropic effects, 
e.g. if the selection is orientation-dependent, anisotropy 
would be introduced f or g alaxies that are aligned by the 
large scale tidal fields [S^ . 

From the observational side, the existence of cosmic 
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dust has recently been detected by [l[ : by using a quasar 
sample at z > 1 and a galaxy sample at z '~ 0.3 from the 
SDSS, the authors find a positive correlation between the 
redness of the background quasars and the overdensity of 
the foreground galaxies, up to an angular separation of 
^ 2° or, a corresponding projected distance separation of 
^ 20/i^^Mpc at z ~ 0.3, which indicates the existence of 
dust correlated with galactic haloes and the LSS. From 
the brightness of these quasars, the authors also find that 
the extinction by cosmic dust occurs at a level compara- 
ble to the magnification by gravitational lensing. Hence, 
it is important to take into account of the effect of dust 
extinction and evaluate its significance for studies of LSS. 

In this paper, by using the results from we inves- 
tigate the effect of dust extinction on the galaxy corre- 
lation function, focusing on the anisotropic features it 
induces, and study how this affects cosmological probes 
through measurements of the galaxy correlation function 
such as the Baryon Acoustic Oscillations (BAO) and the 
linear redshift distortion, e.g. js^-lHI- Our calculation 
accounts for distortion from extinction, as well as that 
from peculiar motion and lensing. 

The outline of the paper is as follows. In ijll] we de- 
rive the formulas used for our calculation, with technical 
details relegated to the Appendices. In i jllll we present 
and discuss the anisotropic features caused by dust ex- 
tinction, focusing on a comparison with those by gravi- 
tational lensing. The effects of dust extinction on cosmo- 
logical probes such as BAO and linear redshift distortion 
are studied in ijlVI Finally, we conclude in |jVl 



II. CALCULATION OF THE DISTORTION 



A. Fluctuation of Dust Extinction 

The optical depth due to dust extinction to a source 
at a comoving distance x away and angular position 0, 
and at observed wavelength Aobs, takes the form: 

Jo (1 + ^) 

where pd is the proper mass density of dust, and z' is 
the redshift associated with the comoving radial distance 
x' on the light cone, (hereafter, we would use z, x inter- 
changeably.) Here / is the extinction efficiency - more 
precisely, pdf gives the inverse (proper) mean free path 
of photons scattered by dust. For simplicity, we assume 
the intrinsic dust extinction properties do not fluctuate 
spatially - namely, / is a constant at fixed rest-frame 
wavelength. Thus fluctuations in the optical depth arise 
entirely from fluctuations in dust density. Subtracting 
the mean optical depth from the above expression, we 
have 

<5r(x, 0; Aob.) = £ j^^^Mx')Sd{x'. 0)fix'. Aobs),(2) 



where Pd(x') is the mean dust density at x' and S^ix' , 
is the fractional overdensity of dust. We will discuss how 
the evolution of pd is modeled below. See [42| for a more 
detailed discussion. 



B. Corrections to the Correlation Function 

When the effects from peculiar velocities, gravitational 
lensing and dust extinction are all taken into account, 
we flnd that, to first order in perturbations, the observed 
galaxy overdensity Jobs is related to its intrinsic counter- 
part 5g by the following 

6ohs = 5g + 5y + 5^ + 5e, (3) 

where 5^, 5^, and 6e are corrections from the peculiar ve- 
locity, gravitational lensing and dust extinction respec- 
tively, and they are given by 



S., = - 



(1 + z) dvi\ 

[5s(z) - 2]k, 
-2.5s(z)5t, 



(4) 

(5) 
(6) 



where v\\ is the line-of-sight peculiar velocity, positive if 
pointing away from us, H{z) is the Hubble expansion rate 
at the redshift of observation z, k is the lensing conver- 
gence, 5t is, as before, the fluctuation of the extinction 
optical depth, s(z) is the factor that describes how the 
distribution of a galaxy sample is modified by the changes 
in the individuals' fluxes. Note the two terms containing 
s(z) in Eqn ([5]) and Eqn ([6]) have opposite signs, which 
just indicates the opposite effects from dust extinction 
and gravitational magnification. For a sharp faint-end 
cutoff selection of the galaxy sample, s(z) (henceforth 
referred to as the number count slope) is given by 



s{z) = 



d\o[ 



:io nohsjz, < m) 



dm 



(7) 



where nobs(-2, < "ni) is the observed mean of the number 
density of the galaxies that are brighter than magnitude 
TO, and TOinax is the limiting magnitude for the sample. 
For a more general sample selection, the expression for 
s(z) is given in Appendix A, where details of our deriva- 
tion for the above results are presented. We have sup- 
pressed the position dependence of (x, 9) on both sides of 
Eqn also for v\\ , At, and 5t, and the wavelength depen- 
dence of Aobs, the characteristic wavelength of the filter, 
for 5t. Note, in our derivation of we have adopted the 
distant observer approximation and restricted ourselves 
to sub- horizon scales [2l|, [2^, . Throughout this paper, 
we would assume the universe is fiat, and set the speed 
of light c = 1 . 

The observed galaxy correlation function 
^obs(xi,0i;X2,02) =(^obs(xi,0i)<5obs(x2,02)) is thcu 
given as a sum of 16 terms 



eobs(i;2) = ^U(i;2), 



(8) 



3 



where we use 1,2 as shorthands for (xi,^i), (X2,^2), 
"a, 6" stand for any of "g, w, e", and .^a6(l;2) = 
((5a(l)^b(2)). The detailed results are presented in Ap- 
pendix B. Here, we make a further simplification of the 
results: we assume the radial separation between the two 
galaxies is much smaller than their mean comoving radial 
distance from us x (i.e. the distant observer approxima- 
tion), and everything evaluated at Xi, with i = 1,2, can 
be calculated by expanding around x, and keeping only 
the contributions that are lowest order in |xi — x|- In 
the following, we give the contributions to ^obs(l; 2) with 
this simplification. 

First, the intrinsic galaxy correlation function is given 

by 



(2^ 



---)p,,(fc,; 



(9) 



where we suppress the dependence of (1; 2) for ^gg, same 
for the ^obS given below, Xj is the position vector for 
(Xi, di), and Pgg{k, z) is the galaxy power spectrum at z, 
with z the redshift corresponding to x- 

Second, when the effect from peculiar velocities is ac- 
counted for, ^obs has the following corrections 



^gv + ivg - ^fo 



_fk_ 
(2^)3 

(2^ 



^ik(xi-X2; 



ik-(xi-X2^ 



{k- zfPgrn{k,z) (10) 



{k- zfPmm{k,z){n) 



where /d = fniz), with fjj = dhiD/dlna, here D is 
the linear growth factor of matter perturbation, and a 
is the scale factor of the universe, fc, z are unit vectors 
pointing respectively in the direction of k and to the cen- 
ter of the galaxy sample, and Pgm,Pmm in turn are the 
galaxy-matter power spectrum and matter power spec- 
trum. Note we have used the plane-parallel approxima- 
tion, which assumes the line-of-sight to all the galaxies 
are the same, parallel to z. The results given here agree 
with the literature on the famous Kaiser's effect j2a|" 
the distortion by coherent bulk motions on large scales 
(in the linear regime), and we neglect the fingers-of-god 
effect |44|- the distortion by random motions within col- 
lapsed haloes (on small scales). For convenience, the sum 
of ^gg I ^gv + ivg and can be calculated equivalently 
by using the formulas given in or [2lj. 

Third, when magnification bias is in addition included, 
^obs bas the following extra corrections 

+ = ^a„,ff2(5s-2)(l + z)|x2-Xi| X 



d^k^ 



.g»k,-x(ei-e2)p^^^^(fc^^^)^ (12) 



-a„Ho'(5s-2 



dxx^ 



1-^ 



(1 + ^)' 



(Pkj_ 
(2^ 



\ X. 

e^'^--^(^^-^^'P,„™(fci, 2)^(13) 



where fl„i is the matter density parameter, Hq is the 
Hubble constant, s = s(z), and k_L is the component of k 
transverse to z. We have used the Limber approximation, 
which makes ^t,^ and ^^^y vanish. 

Finally, with dust extinction, the following terms 
should be added to ^obs 

ige+ieg = -2.5s(l + z)" Aobs) X 

J^e^''^-^'^^-''^Pg4k^,z), (14) 

where Pgd is the galaxy-dust power spectrum. The de- 
pendence of (1 -I- z)~^ (the scale factor at z) comes from 
converting comoving distance to proper distance in the 
calculation for St. As before, we have used the Limber 
approximation, which also makes and ^ev vanish. We 
would neglect the corrections from (or ^ep.) and ^ee, 
the rationale being that extinction is a relatively small 
effect, and so these corrections are small compared to the 
corrections we are keeping, i.e. ^ee *C £,ge and ^ Cms- 
With our approximation and the symmetry these 
terms exhibit, their dependence on (1;2) can be sim- 
plified, which we summarize as follows: besides z, ^gg 
depends only on the distance between the two points 
r = |xi - X2I, Cmm and (Cge + Ceg) depend only on the 
transverse separation Sx± = |x(^i ~ ^2)!, while all oth- 
ers depend on both Sx± and the line-of-sight separation 
<5x = Ixi ~X2|- In particular, note how the magnification 
and extinction distortions have different shapes: ^g^-l-^^g 
exhibits the characteristic lensing- induced linear scaling 
with the line-of-sight separation Sx^ while ^ge + ^eg does 
not depend on it at all. We assume constant galaxy bias 



bg when calculating Pgg and Pgm, i.e. P, 



Pg 
by 



bgP 



gg 



b^P 

^g^ mm 



and 



and we use the transfer function given 
and the non-linear prescription given by [46| to 



calculate 



C. The Extinction Corrections 

To calculate the extinction corrections, we make use 
of the recent SDSS observational results by [l|, where a 
positive correlation between the color (g-i) excess of the 
background quasars and the angular overdensity of the 
foreground galaxies is found, up to an angular separation 
of c:i 100', which suggests the existence of cosmic dust, 
and by using an extinction curve given by the functional 
form of 47| with Ry = 3.1, the one for the standard in- 
terstellar dust in the Galactic disk, the result is converted 
to the following extinction-galaxy cross-correlation 



(Av(0i)<5f (02)) 



2.4 X 10" 



\0^ — 02 



-0.84 



(15) 



where Ay is the V-band extinction from the dust between 
the observer and the quasars, and Sg^ is the angular (2D) 
overdensity of the galaxies. 
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With our formulation, the correlation between Ay and 
Sg^ can be calculated by 



2.5 

In 10 JO 
n{z) f (Pk 



dz{l + z) ^pd{z)f{z,\v) X 



^g,k,.x(ei-e.)p_^^(fc^^^)^ (16) 



v?D J (27r) 

where Zq is the redshift of the quasars, n{z) is the red- 
shift distribution of the galaxies with the normalization 
of fi?^ , their mean surface density, and we have used 
the Limber approximation. Note Ay is defined as Ay = 
2.5Tv/lnlO. In [l|, the galaxies' redshift distribution 
peaks around the mean redshift at = 0.36, hence we 
would make the approximation of n{z)/n^^ — > 5{z — z*). 
Considering Zq > z*, we get 

{Av{0^)5l^{e^)) 



2.5 
In 10 



(27r 



(1 + z*) ^Pd{z*)f{z^,\v) X 



(17) 



where x* is the comoving radial distance corresponding 
to z*. 

Comparing Eqn (jl7p with Eqn (|14p , we find when z = 
z», and when Aobs = Ay, 

^ge +^eg = - In lOs ( Ay (^i )5f (02 )) ■ (18) 

Thus, the result of [l[ can be directly translated into 
the quantity we are interested in. Of course, this direct 
translation works only when the mean properties of the 
galaxies (e.g. redshift, clustering bias and so on) coin- 
cide with those used in [l[. Wc thus need to extrapolate. 
First, we extrapolate in angle. Keeping everything (e.g. 
redshift and so on) fixed, the result summarized in Eqn 
(fT5|) is applicable for angular separations up to 100'. Be- 
yond that, wc assume the shape of the two-point function 
follows that of matter, in other words that 



[£,ge + ieg]{^- 



(Pk 



(19) 

with a normalization that matches the observed result 
at 100'. Next we extrapolate in bias. Since the galaxy 
sample used in [l[ has a clustering bias ~ 1, we multiply 
Eqn (|18p by bg to obtain + S,eg for a different galaxy 
sample with a mean bias of bg. Lastly, we extrapolate 
in redshift. Based on Eqn ([T4| . we assume the following 
redshift scaling for a fixed comoving transverse separa- 
tion 5x^: 



[ige + ieg]{z) « (1 + z)-^ pd{z)D{zf 



(20) 



The number count slope s and the clustering bias hg 
should be redshift dependent as well - we will system- 
atically vary these two parameters in the following com- 
putations to illustrate the range of possibilities. It should 



be stressed that the scaling should also take into account 
of the redshift dependence of the clustering bias of dust 
and the extinction efficiency / for a given Aobs (in our 
case, Aobs = Ay). In this paper, without detailed mod- 
eling of clustering and extinction properties of the dust, 
we would simply neglect the z-dcpendence of both quan- 
tities, (we notice that, for /, this is supported by the 
slowly- varying extinction curve in the visible ran ge, if 
the dust is like that in the Galactic disk, see e.g. j47|.) 
and hope our procedure of systematically exploring vari- 
ations in s and bg is sufficient to bracket the range of 
possibilities. 

For the redshift dependence of pd, we follow [l^, and 
assume the dust particles are ejected to the intergalactic 
medium with a constant yield when new stars are born, 
so 



Pd{z) oc (1 



PsFK{z')dz' 
{l + z')H{z'y 



(21) 



where we set the star formation beginning at redshift 
Zs = 10 [i^l, and for the cosmic star formation rate (SFR) 
PSFR, the mass of baryons that form into star per unit 
como ving volume per unit proper time, we use the results 
from [48| after converting to our cosmology according to 

m. 



III. THE ANISOTROPIC DISTORTION 

In this section, we show our results on the extinction 
distortion of the galaxy correlation function, and com- 
pare it with other anisotropic distortions. Wc first list the 
values of the parameters used in our calculation. Then by 
plotting contours of the galaxy correlation function, we 
explicitly display the anisotropic distortions. The results 
are then extensively studied in the paragraphs following 
the italicized titles. 

The corrections from dust extinction depend on the 
properties of galaxy sample through three parameters. 



An 



s and ba- Wc have set A 



obs 



Ay for the calcula- 



tion in this paper, and for the purpose of illustration, we 
set s = 1.5 and bg = 2, similar to those for the SDSS 
Luminous Red Galaxies (LRGs) [H^l. We discuss the de- 
pendence of the results on these parameters at the end 
of this section. For the cosmological model, we choose it 
to be the best-fit flat ACDM model from the WMAP 7- 
year results that has: matter density f2,„ = 0.27, baryon 
density Vli, = 0.045, Hubble constant h = 0.71, normal- 
ization of the power spectrum erg = 0.8, and the spectral 
index = 0.96 [HI. 

In Figure [U we show the contours of galaxy correla- 
tion function after dividing by 6^ at z = 0.36. The dotted 
lines in the upper three panels are for the intrinsic galaxy 
correlation function ^gg, while those in the lower three 
panels include redshift distortion as well. In each row, 
from left to right, the solid lines are the results with dust 
extinction, magnification bias, and both dust extinction 
and magnification bias added in, compared to the dotted 
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FIG. 1: Contours of the galaxy correlation function divided by at z = 0.36. 5x and Sx± are the line-of-sight and transverse 
separations respectively. Dotted lines in the upper three panels are for the intrinsic galaxy correlation function, while those in 
the lower three panels include redshift distortion as well. In each row, from left to right, the solid lines are the same with the 
dotted lines except that dust extinction, magnification bias, and both dust extinction and magnification bias are respectively 
added in. The colors from black through the rainbow to red represent contour levels at ( — 1.5 x 10^'*, 0, 0.001, 0.002, 0.005, 0.01) 
in the upper three panels, and (-0.002, -0.001, -0.0005, 0, 0.001, 0.002, 0.01) in lower three panels. 



lines. So by comparing the solid lines with dotted lines 
in the left two panels, we can see distortions caused by 
dust extinction to the intrinsic galaxy correlation func- 
tion with redshift distortion included (lower panel) or 
not (upper panel), similar for the middle and the right 
two panels, where the distortions caused by magnifica- 
tion bias and the combined effect of magnification bias 
and dust extinction can be seen. 

The Extinction Anisotropy. The upper left panel ex- 
plicitly shows that dust extinction introduces anisotropy 
to the galaxy correlation function in the {6x±, Sx) plane, 
as is expected from our calculation: Eqn ((Ti)) depends 
only on 6x±, as a result of Limber approximation. For 
a given 6x± , since the amplitude of the intrinsic correla- 
tion generally decreases when 6x (or r) increases, while 
the extinction corrections remain the same, this leads 
to a bigger effect of dust extinction at a larger 5x (or 
r). Similarly, for a given r, since the amplitude of the 
extinction corrections generally increases when Sx± de- 
creases, while the intrinsic correlation remains the same, 
a larger effect of dust extinction is expected at smaller 
6x± , or when the alignment of the separation is closer to 
the line of sight (LOS). Therefore, the distortion by dust 
extinction is expected to be most significant at large sep- 
arations along the LOS, which agrees with our findings 



in the upper left panel of Figure [TJ 

Comparison with Magnification Bias. The upper mid- 
dle panel of Figure [1] allows us to make a comparison of 
the extinction anisotropy with that from the magnifica- 
tion bias, which has been well studied by earlier works, 
e.g. [13, [IHl- Same as dust extinction, the effect of mag- 
nification bias is also more important when the (LOS) 
separation is larger and the alignment of the separation 
is closer to the LOS. In a similar way as before, this 
can be understood by the following: for a given 6x±, in 
general, the amplitude of the total corrections from mag- 
nification bias increases when Sx increases, remains 
the same, -I- increases linearly with 6x,) while 
that for the intrinsic correlation decreases, leading to a 
bigger effect of magnification bias at a larger dx] for a 
given r. the amplitude of the total corrections generally 
increases when Sx± decreases, while the intrinsic corre- 
lation remains the sarne, leading to a bigger effect at a 
smaller Sx±, see also (34. Issf. However, different from 
dust extinction, the anisotropy from magnification bias 
has opposite features — the contours are distorted to the 
opposite sides of their intrinsic locations. 

An examination of Eqn p2|) and Eqn ([14]) tells us that 
(Gm +^ms) ^^^'^ (Cffe +Ceg) havc similar expressions: both 
are given as a 2D integral of the matter power spectrum 
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FIG. 2: Same as Figure \T\ but for z — 2. The colors from 
represent contour levels at (-3 x 10~*, 0, 0.0003, 0.0005, 0.001, 0.003) in 
(-0.001, -0.0005, -0.00025, -0.00015, 0, 0.0003, 0.0005, 0.002) in lower three panels. 



black through 
the upper 



the rainbow to red 
three panels, and 



(note we assume constant bias for both the galaxies and 
the dust) multiplied by a factor. This is the result of 
the fact that both k and St are weighted integrals of 
fluctuations along the LOS, and that we adopt Limber 
approximation in the calculation. There arc two differ- 
ences between the two: first, magnification bias has the 
characteristic linear dependence on the LOS separation 
Sx from lensing; second dust extinction has a dependence 
on the observed wavelength. These differences can be ex- 
ploited to separate out the two effects from data. 

For magnification bias, the sign of the overall multi- 
plicative factor is determined by (5s — 2), while for dust 
extinction, it is always negative. With our choice of 
s = 1.5, (5s - 2) = 5.5 > 0, so (^g^ + ^^g) and (^ge + Ceg) 
always have opposite signs in our calculation. Since the 
correction from is less important at a lower redshift 
compared with those from (^g^ -I- ^^g) [sH], the analysis 
here explains the opposite anisotropic features from mag- 
nification bias and dust extinction throughout the dis- 
played regions in Figure [TJ Specifically, with our choice 
of cosmology, the 2D integral of matter power spectrum 
is positive when Sx± <, 115/i~^Mpc, and negative oth- 
erwise. It is the same for the sign of (^g^i -I- ^^g), while 
the opposite holds true for that of (^ge + ^eg)- There- 
fore, the contours are distorted to where ^gg has larger 
(smaller) values by dust extinction (magnification bias) 
when Sx± <y 115/i^^Mpc, and to where ^gg has smaller 



(larger) values when 6x± ^ 115h ^Mpc. 

Combination with Magnification Bias. Due to the can- 
celing effect between dust extinction and magnification 
bias in our calculation, we can see from the upper right 
panel that the anisotropy from the combination of these 
two is weakened to some extent, with the final features 
dominated by those from dust extinction. 

Including Redshift Distortion. Since in redshift space, 
the most significant anisotropic features in the galaxy 
correlation function are caused by redshift distortion, we 
replot the upper three panels of Figure [T] in the lower 
three ones by including redshift distortion in all the con- 
tours, for a more realistic view of the anisotropics we 
saw before. According to the prediction of the Kaiser's 
effect, which is represented by the dotted lines in the 
lower three panels, the galaxy correlation function has a 
qua drup ole and a hexadecapole component of anisotropy 
j2ll [27t . with the magnitudes controlled by the linear 
redshift distortion parameter (3 = f /bg. Comparing the 
dotted lines with the solid lines, we can see that, same 
as before, dust extinction distorts the contours to where 
they had larger values for most of the displayed regions, 
while magnification bias docs the opposite, so the distor- 
tions from their combination are reduced, with the final 
results still dominated by dust extinction. Along each 
contour, dust extinction would be more important where 
dxj^ is smaller, while magnification bias would be more 
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important where both 5x is larger and Sx± is smaller. 

Dependence on Redshift. To see the redshift depen- 
dence of the anisotropic features, we show the above con- 
tours for a higher redshift z = 2 in Figure [5] For dust 
extinction, its relative importance to the intrinsic corre- 
lation scales with redshift approximately by a factor of 
(1 -|- z)~^pd{z), see Eqn (PH)) . which, according to our 
model for the evolution of dust, increases with redshift 
until z ~ 1.2, where it has a ~ 60% increase from its 
value a.t z = 0.36, then decreases gradually. At z = 2, 
the factor is ^ 30% larger than at z = 0.36, so the extinc- 
tion anisotropy shown in the upper left panel of Figured] 
gets slightly stronger. 

At the same time, the relative importance of mag- 
nification bias to the intrinsic correlation also becomes 
larger, as can be directly seen from the much stronger 
anisotropy shown in the upper middle panel. Both the 
corrections from (^g^ + ^^g) and that from become 
more important at a higher redshift: for the former, its 
ratio to scales with z roughly by (1 -|- z), hence in- 
creases with z; for the latter, the ratio also increases, 
for increases as a sum along a longer LOS, while ^gg 
decreases. The latter also becomes more important com- 
pared to the former, see [11], so, different from at the 
low redshift, the total corrections for Sx± ^ 115/i~^Mpc 
at z = 2 is now also positive, and all the displayed con- 
tours are distorted to where ^gg has smaller values. At 
this high redshift, the combination of dust extinction and 
magnification bias is absolutely dominated by magnifica- 
tion bias, with its effect mildly weakened by dust extinc- 
tion when Sx± <^ 115/i~^Mpc, while strengthened when 
6x± 115/i^^Mpc. Similar results hold when redshift 
distortion is included. 

Dependence on Galaxy Sample. Finally, we discuss 
the dependence of the anisotropic features on the prop- 
erties of the galaxy sample. For dust extinction, the 
anisotropy depends on the ratio of s to bg, and is stronger 
for galaxy samples with larger s or smaller bg. For mag- 
nification bias, the anisotropy is controlled by the factor 
of (5s — 2)/bg, which vanishes for s = 0.4 where (5s — 2) 
switches sign. In our calculation, had we chosen s < 0.4, 
(Gm + ^Ms) would have the same sign as {^ge + £.eg), and 
the combination of magnification bias and dust extinction 
would instead add to each other at the low redshift. The 
anisotropy from redshift distortion is known to be con- 
trolled by P that depends on the sample through 1/bg. 
So for galaxy samples with different s and bg, the rel- 
ative importance of these anisotropic effects would be 
different. Besides s and bg, the anisotropy from dust ex- 
tinction in addition depends on the bandpass used to ob- 
serve the sample. The shorter wavelength the bandpass 
allows, the more extinction the dust particles cause, and 
the stronger the anisotropy would be. Altogether, the 
different dependence on the sample parameters of these 
anisotropic effects provide an opportunity to potentially 
isolate them from one another. 



IV. EFFECTS ON COSMOLOGICAL PROBES 

In this section, we study the effect of dust extinction on 
cosmological probes through measurements of the galaxy 
correlation function. We consider two of these: one is the 
BAO peak, which serves as a standard ruler and probes 
the geometry of the universe, the other is the linear red- 
shift distortion parameter /3, which directly probes the 
growth rate of the universe's structure. 



A. The BAO Peak 

The BAO peak has been detected from the monopole of 
the galaxy correlation function, i.e. the average over the 
alignment of the separation vector, see e.g. 0, 0, BUS]- 
In Figure [3l by identifying the BAO peak as a local max- 
imum in the monopole of ^obs, we show the fractional 
shift of the peak location ArBAO /^bao caused by dust 
extinction (the solid lines), and for comparison, by mag- 
nification bias (the dotted lines) and the combination of 
the two (the dashed lines). The results are given for three 
different redshifts: z = 0.36 (upper panels), z = 1 (mid- 
dle panels) and z = 2 (lower panels) . In Figure |3(a) we 
vary s and keep bg = 2, while in Figure |3(b)[ we vary 
bn and keep s = 1.5. Note redshift distortion is included 



jg and keep s 

when we calculate all the monopoles, which by itself does 
not shift the monopole BAO peak according to the pre- 
diction of the Kaiser's effect [II [H, H^]. 

The scale-dependent correction to the monopole from 
dust extinction shifts r^AO to larger values. This is un- 
derstandable, since, as a negative but increasing compo- 
nent, the correction would shift the local maximum to the 
right. The shift is larger when s is larger or bg is smaller, 
consistent with our analysis above for the extinction dis- 
tortion. As a positive but decreasing component, the 
correction from magnification bias shifts tbao to smaller 
values, except at low redshift and when s < 0.4, and 
the shift is larger when |5s — 2| is larger or bg is smaller. 
Dust extinction tends to act in the opposite direction, 
so the combination of dust extinction and magnification 
bias helps to reduce the shift in r'BAOj except at low red- 
shift and when s < 0.4. For most of the cases, the much 
stronger redshift evolution of the effect from magnifica- 
tion bias, compared to that for dust extinction, makes the 
latter negligible at high redshift, though it is the domi- 
nating effect at low redshift. However, the fractional shift 
of J^BAO by dust extinction is on the order of 10~*, which 
makes it unlikely to be an important factor for probes of 
the monopole BAO peak even at low redshift. 

There has been a claimed detection of the BAO peak 
from the LOS galaxy correlation function by [5^, which 



was disputed by [53[. The issue is subtle: [5j| showed us- 
ing a large set of simulations that even for the monopole 
correlation function, the BAO peak detection is only 
marginal; yet, useful cosmological constraints can be in- 
ferred once combined with external data. Indeed, the 
LOS measurements of 1501 and [531 are consistent with 
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FIG. 3: Fractional shift of the BAO peak from the monopole of the galaxy correlation function by dust extinction (the solid 
lines, labeled by "e"), magnification bias (the dotted lines, labeled by "/^") and their combined effects (the dashed lines, labeled 
by "e + n"). In Figure 3(a) we keep bg at our default choice of 6g = 2 and vary s, while in Figure 3(b) we set s at the default 
value of 1.5 and vary bg. From top to bottom, the three panels in both figures are for z = 0.36, z — 1 and z = 2 respectively. 
Note redshift distortion is included in the calculation for all the monopoles, which by itself does not shift the monopole BAO 
peak according to the prediction of the Kaiser's effect [2ll. [2^. [27| . 



each other. It is the interpretation (of how the data 
should be used) that differs. For our purpose in this 
paper, it suffices to note that the dust extinction correc- 
tion has no LOS dependence and so would not shift the 
LOS BAO peak at all. 

B. Redshift Distortion 

According to the prediction of the Kaiser's effect 
[21I [26i . ,27, ] , the redshift distortion parameter f3 can be in- 
ferred from the anisotropy of the galaxy correlation func- 
tion through 

eoW-CoW l-l-|/3+i/32' ^ ' 

where ^0 f^nd ^2 are the monopole and quadrupolc of ^obs, 
and ^0 is the volume average of ^q, given by 

Ur) = ^ f WV^dr'. (23) 

However, when the effects of dust extinction and magni- 
fication bias arc taken into account, their corrections to 
the monopole and quadrupole would shift the inferred (3 
from its true value, and moreover, the scale-dependence 
of the corrections may cause the inferred /? to be scale- 
dependent too. This is what we find in Figure |4l where 
the fractional changes in /3 caused by dust extinction 
(the solid lines), magnification bias (the dotted lines) 
and their combination (the dashed lines) are presented 
at z = 0.36 (the upper panel), z = \ (the middle panel). 



and z = 2 (the lower panel) respectively. The results are 
obtained with our default values: s = 1.5 and bg = 2. 

As can be seen, the changes by dust extinction and 
magnification bias are in opposite directions, with /3 get- 
ting bigger for the former and smaller for the latter, so 
the changes by their combination tend to be reduced. If 
we define Q = £,2/(^0 ~ Co)j we find dust extinction in- 
troduces A(^o — 63) > (A^o increases with r), A^2 < 
(A^ increases when the alignment of the separation is 
closer to the transverse) , while with the prediction of the 
Kaiser's effect, (^0 — £,0) smd ^2 ai'c both negative (same 
as {£,gg — ^gg), see e.g. [l^]), so dust extinction leads to 
AQ > 0. When we infer from Q through Eqn ((22|) . 
we have dQ/dj3 > 0, so this explains why dust extinc- 
tion causes (3 to be larger. With s > 0.4, magnification 
bias introduces opposite changes, so the smaller value of 
the inferred /3 when magnification bias is included can be 
understood in a similar way. 

Our calculation also shows that the fractional changes 
by dust extinction are on the order of a few percent, and 
vary mildly with redshift, while those by magnification 
bias are on the level of ^ 1% at z = 0.36, but can grow up 
to 40% at 2 = 2, which turns it from a sub-dominant 
effect at low redshift to a dominating effect at high red- 
shift. These fractional changes indicate that both dust 
extinction and magnification bias would be non-negligible 
for redshift distortion probes for the purpose of precision 
cosmology, e.g. |38l - l41| . especially at low redshift for the 
former and at high redshift for the latter, though the 
canceling effect between these two helps to reduce these 
systematics. 

Finally, we point out that the scale dependence of the 
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FIG. 4: Fractional changes in the linear redshift distortion 
parameter /3, inferred through Eqn (|22p . as a function of the 
distance separation r. The solid lines show the changes caused 
by dust extinction (labeled by "e"), the dotted lines are those 
by magnification bias (labeled by "^"), and the dashed lines 
are by their combination (labeled by "e + /i"). To see the 
changes more easily, we also show A/3 = using dot-dashed 
lines. From top to bottom, the three panels are for z = 0.36, 
z — 1 and 2 = 2 respectively. The values for s and bg axe set 
at our default choice. 



inferred /3 may be mistakenly interpreted as an indication 
for modified gravity, which, different from General Rel- 
ativity (GR), can give a scale-dependent growth factor, 
and lead to wrong conclusions for tests of GR through 
probes of the redshift distortion parameter [55l - [58l |. if 
these systematics are neglected. 



V. DISCUSSION 

Inhomogeneities in the extinction of the galaxies' fluxes 
by cosmic dust, whose existence is recently detected by 
[l| , modify the distribution of a flux-selected galaxy sam- 
ple. This is similar to what the inhomogeneous magnifl- 
cation of the galaxies' fluxes by gravitational lensing does 
to the galaxies' distribution, but with opposite effect on 
average. Hence, in addition to the Alcock-Paczynski ef- 
fect, redshift distortion, magnification bias, dust extinc- 
tion is a fourth effect that would create an anisotropic 
distortion to the galaxy correlation function for a flux- 
limited selection. In this paper, we have studied this 
extinction distortion to the galaxy correlation function, 
and evaluated its effect on cosmological probes such as 
the BAO and the linear redshift distortion. 

We use the extinction-galaxy cross correlation found 
by [l| to calculate the corrections to the galaxy correla- 
tion function from dust extinction. We extrapolate their 
results to larger scales by assuming dust traces galaxies. 



and to other redshifts by assuming the evolution of dust 
follows that of the stars. With the choice of Aobs = Ay, 
s = 1.5, 6g = 2, we show the anisotropic extinction dis- 
tortion together with that from magnification bias and 
redshift distortion in Figure [T] and Figure [51 

We find the distortion by dust extinction alone is most 
significant along the LOS and at large separations, which 
is similar to that by magnification bias. Their precise 
shapes arc different though. Lensing induces a correction 
to the correlation fimction that rises with the LOS sep- 
aration, while extinction does not. The correction from 
dust extinction depends only on the transverse separa- 
tion 5x± , and with our choice of cosmology, it is negative 
when dx± < 115/i~^Mpc, positive otherwise. With our 
choice of s > 0.4, the correction is almost always opposite 
in sign to that from magnification bias, leading to the op- 
posite anisotropic features seen in the distortions by these 
two effects. So, the distortion by their combined effect 
tends to be reduced. The extinction distortion evolves 
with redshift approximately by (1 -I- z)~^ pd{z), which is 
much milder than the evolution of the lensing distortion. 
At low redshifts (z <^ 1), the extinction distortion tends 
to be more important than lensing, while the opposite is 
true at high redshifts. 

By identifying the BAO peak as a local maximum, we 
find the scale-dependent correction from dust extinction 
to the monopole of the galaxy correlation function shifts 
the monopole BAO peak to larger scales, but the shift 
is on the order of 10"'^, and it does not change much 
when varying s, bg and z. At the same time, the scale- 
independent correction from dust extinction to the LOS 
correlation function evaluated at a fixed 6x± docs not 
shift the LOS BAO peak at all. So for probes of the 
BAO, dust extinction is probably a negligible effect. 

The anisotropic extinction distortion also introduces 
biases in the linear redshift distortion parameter 13, in- 
ferred from the monopole and quadrupole of the observed 
galaxy correlation function according to the prediction of 
the Kaiser's effect. We find with dust extinction, the in- 
ferred (3 is bigger than the true value by up to a few 
percent, and the shift varies mildly with redshift, while 
with magnification bias (s > 0.4), /3 is smaller than the 
true value by up to percent level at low redshift {z <i 1), 
but to ~ 40% at high redshift. This suggests both ef- 
fects are non-negligible for precision probes of /3, espe- 
cially for extinction at low redshift and lensing at high 
redshift, though their combination tends to reduce the 
overall shift in /3. With these two effects, the inferred (3 
also becomes scale-dependent, which should be taken into 
account for tests of GR through the scale-dependence of 
/3 (the growth factor). Our analysis on /? can be extended 
to Fourier space. It is possible that the changes in /3 in 
Fourier space would be smaller, as suggested by earlier 
works of [3a |. who find that, for magnification bias, its im- 
pact on galaxy correlation is less severe in Fourier space 
than in real space. However, we would leave a rigorous 
study for future work. 

The extinction distortion (extinction correction nor- 
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malized by the intrinsic galaxy correlation) scales with 
the properties of the galaxy sample as s/hg, so it is more 
significant for galaxy samples with larger s or smaller 6^, 
and it also depends on Aobs , the bandpass used to observe 
the sample, shorter Aobs, stronger distortion. This is dif- 
ferent from the distortion by magnification bias, which 
depends on (5s — 2)/6g, and has no dependence on Aobs- 
Moreover, these two distortions have different shapes, 
with lensing exhibiting the signature linear dependence 
on the LOS separation, while extinction depends exclu- 
sively on the transverse separation. These differences can 
be used to separate the two effects, to allow a simultane- 
ous study of both the cosmic extinction and the cosmic 
magnification [59| . which we hope to explore in future 



work. 
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APPENDIX A 

In this Appendix, we give our derivation for the observed galaxy overdensity in the presence of peculiar velocities, 
gravitational lensing and dust extinction, as given by Eqn in i jll Bl 

Suppose we are making observations in a smooth universe without dust, and galaxies can be thought of as test 
particles that are at rest on the comoving grid, the number of galaxies we observe whose positions are within the 
comoving volume element around {xg, dg) with a radial extent dxg and an angular extent cPOg, and whose fluxes are 

within the range of [/g", fg" + dfg^] is ^g{xg,Og; fg^)xldxg(POgdfg^ . Here, the observed flux fg is labeled by the 
source rest- frame wavelength Xg , while its dependence on the observer rest-frame wavelength Aobs is omitted, because 
in the following, we are about to compare this hypothetical observation with the real observation, while Aobs is the 
same for the both (same filter), Xg is not. Note, we use the subscript "g" to label the quantities from our hypothetical 
observation, and throughout this paper, we assume a flat universe. 

In the real universe with perturbations and dust, the observed coordinates and fluxes of these galaxies will be 
shifted according to the following 



Xg -> X = Xg^ 

Og ^ e = eg + seg 

fg' ^ /' = e-M/g^ 



{l+Zg)Vl\ 



H{Zg) 



(24) 

(25) 
(26) 



where the change in Xg follows from that in the redshift Zg (= Aobs/Ag — 1), which is a direct observable and shifts as 



Z = Zg + (1 + %)U||, 



(27) 



where we have set the speed of light c = 1, kept only to the first order of perturbations, as we do throughout the paper, 
and we have included only the change caused by the line-of-sight peculiar velocity of the galaxies W|| ; the apparent 
angular position is displaced by gravitational lensing by an amount of 59 g] the observed flux is changed in two ways: 
first, it is magnified by gravitational lensing by a factor oi A = det[90g/90^]~^, and second, it is reduced by dust 
extinction by a factor of e""^, with t the extinction optical depth. For additional effects that affect the galaxies' flux, 



see [60|-|62[ . We have suppressed in the above equations the position dependence of {xg^ 6g) for f ||, 56g, r, and A, and 



also the wavelength dependence of Aobs for r. 
should be conserved, so we have 



Despite the changes of these observables, the number of the galaxies 



$(x, e- f^)x'dxd'edf^ = <fg{xg,e9; fg')xldxg<fegdf^^ 



(28) 



Given the results from a real observation, a galaxy sample can be selected by specifying the selection efficiency 
function e{f^). As an example, for a selection with a sharp faint-end cutoff at the limiting fiux of f^^^, ^if^) is just 
the heaviside step function 8(/'^ — /min)- The observed galaxy density of the sample is then given by 



nix,0) 



e{f^Mx,0;f^)df 



(29) 



Note, hereafter, we would call only the quantities from the real observation the "observed" ones, while those from our 
hypothetical observation (in a smooth universe) would be called the "intrinsic" ones. By using Eqns and 
Eqn dUl), we find 



n(x,0) = (l-2Ac) 



1 - 



(1 + Zg) dv\\ 



H{zg) dxg 



^Af^^)^,{Xg,Og-J^^)df^^ 



(30) 
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where the first bracket on the right hand side of the equation comes from dPOg/dPO = \/A, and we have used A ^ 1 + 2k 
in the weak lensing regime, with k the lensing convergence; the second bracket comes from {Xglxl^dYg/dx, and we 
have adopted the distant observer approximation, also restricted ourselves to sub- horizon scales |2ll . |26| . By defining 
the intrinsic galaxy density to be 



'g 



(31) 



we get 



n(x,6>) = Ugixg^Og) 



H{.Zg) dXg 



(32) 



with Sg defined by 



SgiXg) 



de 



2-5ng{xg) J df 



(33) 



where quantities with overbar represent their mean values. Up to the first order of perturbations, we find Sg{xg) 
six), with 



six) 



1 



2.5n(x)7 df^ 



4^f'Hx-J')df\ 



(34) 



which practically can be obtained from observations. When e(/'*') is a step function, Eqn ([M)) reduces to Eqn ([7|). 
Note m = — 2.51og]^Q /'*' with a constant offset. 

Finally, with the definition for the observed galaxy overdensity Sohsix,^) = in — n)/n and that for the intrinsic 
overdensity Sgixg, Og) = iug — fig) /fig, we obtain our results from Eqn ([32]) 



(I + z) dvii 

Sohs = Sg- ^TTT^^ + [5s(^) - 2]k - 2.5siz)ST, 
Hiz) dx 



(35) 



where instead of ixg, Og), the quan tities on the right hand side are evaluated at (x, 6), same as (5obsj which is accurate 
to first order in perturbations [35|. 



APPENDIX B 



In this Appendix, we give the expressions for the individual ^abS with distinct combinations of "ab" , without keeping 
everything to the lowest order of |xi — x| as done in §11 Bl The calculations of these terms are straightforward, so we 
just present the results here for reference. 

The intrinsic galaxy correlation function is given by 

/d^k 
__^.M^.-^.)p^^^k), (36) 

where Pggik) is the power spectrum for the galaxies at zi and Z2, with Zj the rcdshift corresponding to Xij and we 
do not explicitly specify this redshift dependence, similar for Pg,„(fc) and Pmmik) below. With the plane-parallel 
approximation and the sub-horizon limit of the continuity equation, the corrections from the peculiar velocities are 
given by 

^e^^<-^--^\k-z)'Pg,^ik), (37) 
U(l;2) = /d(1)/d(2) J ^e^'^<-^--^\k-z)^Pr.,^ik), (38) 



where /d(«) = foizi), and fo — din D/ din a. By using the Limber approximation and the Poisson equation. 
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magnification bias in addition gives the following corrections 



e<,^(l;2) = -a„ifo'(5s2-2)(l + zi)e(x2-Xi)xi(l-— ) X 



X2 

/|^e^'^--^(«-«^)p,,„(fc^,zi), (39) 

/o \2 i-min(xi,X2) 

2) = (^2"'"^" j (^^1 " 2)(5s2 - 2) rfx(l + z)\^ X 



(1 - ^)(1 |-%e*->^(«-«^)p„„„(fc^,z), (40) 



Xi X2 y (27r) 

C.A.(1;2) = 0, (41) 

where Si is s{z) for the galaxies at redshift Zi, and is the heaviside step function. Finally, with dust extinction, ^obs 
has the following extra corrections, 

^,e(l;2) - -2.5,S2(l + zi)-Vd(^i)/(^i,Aobs)e(x2-Xi) j |^e'''-'^^(^^-^=)p,,(fci,zi), (42) 
C.e(l;2) - 0, (43) 

where, again, we have used the Limber approximation. Since both and ^ee are negligible, we do not give their 
expressions here. 



